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PREFACE 



The first systematic attempts to apply research and development strategies 
to education occurred with the establishment of research and development cen- 
ters in 1964 and regional educational laboratories in 1965. However, these 
strategies as they applied to education were neither clearly defined nor well 
tested. It has become the task of R & D centers as well as regional labora- 
tories to refine existing strategies on the basis of experience and to invent 
new ones. 

It is appropriate, then, that the Wisconsin Research and Development 
Center for Cognitive Learning and the Southwest Regional Laboratory for 
Educational Research and Development cooperate in presenting strategies cur- 
rently employed at each institution to the scholarly community in a symposium 
at the American Educational Research Association meetings. Professor 
Klausmeier, organizer and chairman of the symposium, outlines R & D strategies 
of the Wisconsin R & D Center in his introduction. Two applications of con- 
trolled experimentation in school settings are described by Dr. Wardrop and 
Mrs. Quilling. Dr. Romberg presents the procedure to be utilized in the 
development and refinement of prototypic instructional systems. Representing 
the regional laboratory approach is Dr. Schutz, who considers strategies for 
improving instruction through product development. Finally, Professor McLean 
reacts to the strategies presented, comparing them with those used at a 
Canadian counterpart to R & D centers . 



James E. Walter 
Director of Dissemination 
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ABSTRACT 



Research and development strategies designed and implemented at the 
Wisconsin Research and Development Center for Cognitive Learning and the 
Southwest Regional Laboratory are discussed in papers by the directors of 
each and by members of the R & D Center staff. These papers further give 
examples of the research and development activities underway in each 
organization . 

Papers on the controlled experiment focus upon its applicability in dif- 
ferent settings. The large-scale experiment involving many classes uses 
the class as the experimental unit, whereas smaller experiments conducted 
in the newly devised Research and Instructional Units may use the individual 
as the unit. The steps in developing and refining a prototypic instructional 
system are clearly outlined in another paper. Finally, the usefulness of 
formative evaluation procedures in effecting educational improvements are 
discussed. 
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RESEARCH AND DEVELOPMENT STRATEGIES IN EDUCATION 



Herbert J. Klausmeier 
Professor of Educational Psychology 
The University of Wisconsin 
Director, Wisconsin R & D Center 



The purpose of this collection of papers , 
as of the symposium from which it developed, 
is to identify and discuss contributions that 
can be made simultaneously to improving in- 
struction and to refining teaching-learning 
theory through the use of relevant research 
and development strategies applied to educa- 
tion . This kind of research and development 
is of very recent origin. The first attempt to 
apply research and development strategies 
systematically to education was initiated by 
the U.S.O.E. in 1964 when research and de- 
velopment (R & D) centers, including the Wis- 
consin Research and Development Center for 
Cognitive Learning, were established on four 
university campuses. As of January 1968, 
there were nine R & D centers . Each center 
has identified a problem area — such as cogni- 
tive learning, educational stimulation in early 
childhood, evaluation, or teacher behavior — 
as its focus . Each center conducts research 
in oider to extend knowledge and theory related 
to the problem area and also engages in related 
development to improve educational practices. 

In 1965 the Elementary and Secondary Edu- 
cation Act was passed. In connection with 
this legislation, 20 regional educational lab- 
oratories had been established as of January 
1968. The regional laboratories are incorpo- 
rated as separate legal entities and are not 
affiliated with institutions of higher learning. 

In general, a regional laboratory engages quite 
heavily in development and development-related 
research while an R & D center has a somewhat 
greater emphasis on basic research and a lesser 
emphasis on development. 

At present then there are 9 R & D centers 
and 20 regional laboratories engaged in re- 
search and development designed to improve 
educational practices . Different from long 



established and well funded research and de- 
velopment in agriculture, space, and industry, 
research and development in education does 
not have clearly defined and well tested strate- 
gies . So the staff of the Wisconsin R & D 
Center is engaged not only in attempting to 
extend knowledge about cognitive learning and 
to improve related educational practices, but 
also in inventing research and development 
strategies in education and refining existing 
ones . 

Before considering some detailed strategies 
employed in the Wisconsin R & D Center for 
Cognitive Learning, its focus end global strate- 
gies should be considered. In the R & D Center 
we differentiate between basic and applied re- 
search in terms of purposes. Basic research 
is conducted to generate knowledge and is not 
concerned with whether the knowledge may be 
useful in improving educational practice. 
Development-based research, referred to by 
some as applied research, is specifically con- 
cerned with developing a substantive or pro- 
cedural output designed to achieve specified 
objectives and with ascertaining how well it 
achieves the objectives under varying condi- 
tions with children or instructional personnel 
of varying characteristics. 

The prevalent form of basic research is the 
controlled experiment and its variants. Most 
controlled experiments designed to extend 
knowledge about learning are conducted in the 
laboratory, or under laboratory conditions. 

Other types of basic research employed in edu- 
cation include correlational, factor analytic, 
survey, and case study. These studies are 
often conducted in the school setting, with 
children, school personnel, or both as subjects. 

The two distinguishing features of develop- 
ment-based research are its purposes , as given 



before, and the procedures employed. Develop- 
ment-based research encompasses identifying 
a deficiency or problem area in some component 
of an instructional system; identifying or formu- 
lating objectives of that component; and devel- 
oping, testing, and refining a new component 
to achieve the objectives . Development-based 
research may be initiated by practitioners or 
by scholars. The need for it is experienced by 
practitioners directly and by researchers through 
formal and informal surveys. Small develop- 
mental projects may be initiated and executed 
by school personnel. Comprehensive projects 
designed to produce an instructional system in 
a subject field at the elementary school level 
may run for five or more years . The testing and 
refinement of a substantive or procedural out- 
put through various revisions may involve con- 
trolled experimentation, correlational studies, 
and case studies. It must involve ascertaining 
how well the output achieves the objectives for 
which it was developed. 

The evaluation of instructional materials , 
procedures , and equipment that have not been 
tested in the school system may be designated 
as a third kind of research. The purpose here 
is to improve instruction, not to add to knowl- 



edge. Again, several forms of research may 
be executed in an evaluative study; however, 
answers must be sought concerning how well 
the product performs in connection with clearly 
stated criteria. 

Figure 1 indicates that basic research and 
developmental research are conducted in each 
of the two major programs of the Center. In ' 
Program 1 basic research is conducted to ex- 
tend knowledge about processes and conditions 
of learning, whereas in Program 2 the basic re- 
search is conducted to extend knowledge about 
instructional processes. The straight line con- 
necting the two sets of basic research activi- 
ties indicates the relationship of similar methods 
and concepts in the two programs. However, 
many variables not considered in learning re- 
search must be taken into account in research 
on instruction , particularly those variables 
associated with teacher behaviors and charac- 
teristics , student behaviors and characteristics , 
subject-matter content and sequence, utiliza- 
tion of time , utilization of space , and organi- 
zation of personnel for instruction. 

The other relations between basic research 
and development-based research shown in Fig- 
ure 1 are of primary interest. Obviously, there 
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Figure 1 . Relationships among research and development activities at the 
Wisconsin Research and Development Center for Cognitive Learning 



(*N 

* 



O 

ERIC 



2 



are many relations, rather than one linear rela- 
tionship. In the A] ■ ► A 2 ► 3 

sequence, basic research in the behavioral sci- 
ences provides the foundation for the develop- 
ment of procedureal and substantive outputs, 
such as instructional materials and equipment 
for use by students or descriptions of proce- 
dures to be used by teachers and other educa- 
tional workers. In turn, these outputs are 
researched and refined with school-age chil- 
dren in laboratory situations or directly in 
school settings. In actual practice in our R & 

D Center there is not a clear-cut terminal point 
at which research results are stated and product 
development begins, nor are there separate 
groups of personnel , one doing research and 
another product development. Rather, the com- 
bined research and development strategy here 
involves cooperative effort by specialists in 
learning, the subject disciplines, and meth- 
odology and also teachers or other school per- 
sonnel in Multiunit Schools. 

The Bj ^ B2 ^3 and the 

Ci— — — C2 3 sequences are 

more typical of present research and develop- 
ment activities at the Wisconsin R & D Center. 
Here one or more investigators with similar - 
interests in either learning or instruction do 
the basic research, develop a substantive or 
procedural outcome intended for students or 
teachers , and carry through with the research 
in school settings. Consultants with various 
specialties and school personnel participate in 
the activities at appropriate points. 

So that specialists in various fields may be 
available to several projects, regardless of 
size and personnel combinations, the Wisconsin 
R & D Center is developing a large technical 
section, staffed by persons of varying special- 
ties who participate simultaneously in several 
projects of both programs. Also, in order to 
secure continuous cooperation with school per- 
sonnel, Multiunit Schools that facilitate re- 
search and development activities have been 
organized. 

The three preceding sequential strategies 
are operative in the Wisconsin R & D Center 
for Cognitive Learning. There are other se- 
quences. Lines Dj and Ej indicate that, al- 
though some basic research may terminate in 
relevant reports, monographs, or books, the 
results may lead directly to developmental re- 
search in school Settings, without the inter- 
mediate step of developing a product to be used 
by children or teachers. Also, lines D2 and E2 
show that some research results may be put 
directly to use in educational improvement. 

Both sequences occur in our Multiunit Schools 
when teachers participate in experiments , se- 
cure feedback about the results, and^put the 
relevant results into practice. 



The preceding discussion has focused on 
ideas and activities moving in the general direc- 
tion of research — — development— 
practice. As shown in Figure 1 by the dotted 
lines, the sequence may be in the other direc- 
tion. For example, the desire on the part of 
school people and others to improve the edu- 
cation of disadvantaged children has led to 
the formulation of more precise educational 
specifications for the subsequent development 
of substantive and procedural products. The 
same practical problem has also generated 
some basic research dealing with cognitive 
skills. The continuous interaction of the 
Wisconsin R & D Center staff with school peo- 
ple, and also the continuous examination of 
educational problems by the R & D staff, hove 
markedly influenced the development activities 
of the Center. At the present time much basic 
research has ueen generated in connection with 
developmental activities. Probably a lesser 
amount of basic research has led directly to 
development. 

The actual sequencing of activities is not 
precise in either direction. Rather, there is 
more a continuous interplay of researching and 
developing. The personnel primarily engaged 
in improving an instructional system conduct 
comprehensive development-based research to 
ascertain how well the program works with the 
clients or consumers and make changes accord- 
ingly. Although not conducting basic research, 
they refer to contemporary theory and also to 
the results of basic research through reading 
and attending conferences and by bringing in 
knowledgeable consultants . Thus , a community 
of scholars representing various specialities, 
some within the Center on a more permanent 
basis and others from outside the Center on a 
consulting basis, is an essential component of 
an effective research and development strategy 
in education. 

in the preceding discussion a numbei of 
important points have been made. First, the 
initiation of activities designed to improve 
educational practice may be at any oi three 
points: an analysis of deficiencies in educa- 
tional practice, the design and development of 
instructional products, or the conduct of basic 
research. Second, the output from basic re- 
search is knowledge, while from development 
and applied research the outputs are instruc- 
tional materials, procedures, organizations, 
etc. that are designed specifically to improve 
educational practice. Third, there are several 
relations among research, development, and 
improved practice rather than a single linear 
research— development ^ 

practice sequence. 

The latter needs special emphasis in view 
of two widely disseminated models of research 

3 



development practice. 

In 1964 Hilgard outlined c. viewpoint of many 
educational psychologists os shown in Fiovrc 

2. In this sequence it is assumed that le.un- 
ing theory has relevance for improving instruc- 
tion but that development activities and applied 
research are essential intervening steps. 
Hilgard points out that the six steps are not 
discrete, but are on a continuum, and that 
invention plays an important role in bringing 
about improved practice. Nevertheless, there 
is a strong tendency toward emphasizing basic 
research as both the inititation point for and 
primary contributor to subsequent development 
and thence improved practice. 

The Guba-Clark model is shown in Figure 

3. This model also puts research, develop- 
ment, diffusion, and adoption in a linear ar- 
rangement from left to right. Guba and Clark 
noted that the model was a unidimensional 
analysis of change roles influenced by many 
variables and also that change did not neces- 
sarily begin with research. Vhile Guba and 
Clark may accept the kind of relations outlined 
in the model of the Wisconsin R & D Center, 
they did not make the same relations explicit 
in their model. Furthermore, there is a wide- 



spread tendency, particularly among behavioral 
scientists interested in education, to assume 
that the primary basis for improved educational 
practice is the results of basic research. 

More specific strategies in research and 
development are presented next. The first 
three speakers in this symposium draw from 
their experiences at the Wisconsin Research 
and Development Center for Cognitive Learning . 
Richard £. Schutz, Director of the Southwest 
Regional Laboratory for Educational Research 
and Development at Io~ Angeles, presents 
their strategies for developing and evaluating 
instructional materials. Leslie D. McLean of 
the Ontario Institute for Studies in Education 
discusses the strategies and techniques pre- 
sented. Let me point out that in this sym- 
posium we are not engaged in painting pictures 
of something that may emerge in the decades 
ahead. The Wisconsin R & D Center and the 
Southwest Regional Educational Laboratory 
may be visited , and our strategies may be 
observed and discussed. We are, however, 
vitally concerned with the future and are thus 
open to suggestions that may help us in devel- 
oping more creative research and development 
strategies to improve education. 
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Figure 2. Steps in research on learning — pure research to technological development. 



Source: Hilgard, Ernest R. A perspective on the relationship between learning theory and educa- 
tional practice. In Hilgard, Ernest R. (Ed.) Theories of Learning and Instruction , Thirty- 
sixth Yearbook. Chicago: National Society for the Study of Education, 1964. 
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Figure 3. A Classification Schema of Processes Related to and Necessary for Change in Education 

Source: Guba, E. G. , & Clark, D. L. An examination of potential change roles in education. Paper read at Symposium 
Innovation in Planning School Curricula sponsored by the National Education Association Committee for Study of 
Instruction, Airielhouse, Va., October, 1965, P. 8. 
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CONTROLLED EXPERIMENTATION IN MULTICLASSROOM SETTINGS 



James L. Wardrop' 

Assistant Professor of Educational Psychology 
Center for lrstr«cti«.al Research and Currici lum Evaluation 
University' of Illinois 



"Research" in education has many faces. 

It may appear as an associationul study, a 
case study, an informal description oi events, 
a controlled experiment, cr any numoer of other 
activities. For the purpose of elucidating 
causal relationships— answering questions cf 
the form, "What is the effect of variable x on 
behavior Y?" (where X and Y are probably both 
multivariate rather than univariate) the appro- 
priate face (or should I say mask?) is the con- 
trolled experiment. 

The phrase "controlled experiment" carries 
with it some obvious connotations: there is 
some independent variable or variaples which 
the experimenter manipulates? there are two 
or more groups wh'ch are exposed to different 
levels of these independent variables; there 
may be an attempt to maxe these groups homo- 
geneous with respect to characteristics, like 
IQ and past achievement, which are not under 
the experimenter's control; there is some de- 
pendent variable which is amenable to being 
measured with sufficient precision that differ- 
ences may be detected when they exist; this 
criterion variable is one which is expected to 
be affected by the experimental manipulations; 
and the criterion scores are subjected to some 
sort of analysis in order to arrive at some de- 
cisions about the effectiveness of the experi- 
mental manipulations. 

In addition to these rather basic character- 
istics , there are some other very important 
aspects to controlled experimentation. Con- 
sider, for example, the objective of eliminating 
as many as possible of the alternative explana- 
tions that might reasonably be offered for the 



* During 1966-1967 Dr. Wardrop was a Title IV 
postdoctoral fellow at the Wisconsin Research 
and Development Center for Cognitive Learning. 



results ct an experiment. Campbell, and Stanley 
(1963) have presented an excellent discussion 
of some of these alternatives — factors which 
might bias the outcome of an experiment such 
as selection, maturation, and statistical re- 
gression. For a thorough discussion of such 
factors, I refer you to the Campbell-Staalev 
monograph. For a good nonstatistical discus- 
sion of experimental design, David Cox's 
Planning of Experiments (1958) has much to 
recommend it. 

In educational research, as in many other 
scientific enterprises , the argument over basic 
versus applied research continues to rage. 

(See, e.g., Cronbach, 1966, and Ebei, 1967.) 
How* applicable, for example, are the results 
of laboratory experiments to the natural setting 
of the classroom? Can the results of basic 
research be applied on a large scale in class- 
room settings ? I am now raising questions 
about the generalizabilitv of results , what 
Campbell and Stanley call "external validity." 
It is at this point that the need for controlled 
experiments in multiclassroorn settings be- 
comes apparent. 

Much of the research that has been done 
in education has been justly criticised because 
statistical analyses have been based on indi- 
vidual students' performance when the appro- 
priate experimental unit was the classroom. 
(See Cox, 1958, p. 2; Lindquist, 1940, pp. 
21-24.) In this connection, it is worth quot- 
ing Lindquist, who said: 

In general. . .many of the samples 
employed in educational research 
consist of a small number of in- 
tact groups (such as classes in 
the same or different schools , 
groups of pupils in separate 
buildings in the same system, or 



groups of pupils from different 
communities or geographical 
regions) , or of u small number 
of subsaniples selected from 
different "strata" in the popu- 
lation, . , .In all such cases, 
the "size" of the sample is 
dependent, not upon the num- 
ber oi individual observations, 
but upon the number of intact 
groups or subsampies of wnich 
the total sample is constituted. 

In other words , the unit of sam- 
pling in educational research is 
often the class, the school, or 
the community, rather than the 
pupil. 

Julian Stanley (1965} madd explicit the fact that 

In determining what the ex- 
perimental unit is, it makes no 
difference whether pupils were 
assigned at random to the class- 
rooms or systematically. The 
basic point is that the _tth treat- 
ment was assigned at random to 
the classroom as a whole , rather 
than just tc certain individuals 
within it randomly. 

The point of all this is as follows: if one 
wishes to determine the effectiveness of some 
curriculum package (which is administered by 
a teacher to a class ) , or of some teaching 
method (again administered by a teacher to a 
class ) , or of group size (where instruction is 
given to small or large groups ) , or of some 
scheduling system (where, for example, groups 
of students are assigned either modular or tra- 
ditional class schedules), one is faced with 
the situation in which the experimental unit, 
and hence the unit of analysis , is a group or 
class of students, not the individual student. 

In attempting to determine the applicability 
of laboratory findings in classroom settings, 
it is imperative for internal validity that con- 
trolled experiments be used, and for both in- 
ternal and external validity that these experi- 
ments be carried out in a number of classrooms 
or schools and the classroom or school be 
treated as the unit of analysis. 

To illustrate the kinds of problems being 
investigated through this particular approach, 
let us consider some specific multiclassroom 
experiments being conducted by the Wisconsin 
R & n Center. One project, "Situational Varia- 
bles and Efficiency of Cognitive Learning," is 
designed to investigate selected variables and 
conditions associated with efficient learning of 
concepts and cognitive skills and to develop 
and test a system to facilitate concept learning. 



a system of motivation, and a system of Indi- 
vidualization in school settings. As a direct 
outgrovrth of an earlier project designed to 
clarify funct onal relationships among variables 
related to efficiency of cognitive learning in 
.’aboratory settings, this project represents a 
direct attemut to validate laboratory findings 
in the clas sroom , thus extending knowledge 
about theoretical relationships and at the same 
time improving classroom instruction. 

Another project which is using multiclass- 
room experimentation deals with Enylish syn- 
tax and composition. This project has provided 
materials for experimentation on learning varia- 
bles in school settings (Blount, Klausmeier, 
Johnson, Fredrick, & Ramsay, 1967) and de- 
veloped instructional materials for use by 
teachers and students in junior high schools. 
Here again the project has the two purposes 
of generating and refining knowledge about 
variables affecting classroom learning and of 
improving instruction in English grammar and 
composition. While the major emphasis in the 
previous project was on theory refinement, the 
main focus of this project is on improving 
classroom instruction. 

In order to illustrate something of the nature 
of multi clas sroom experimentation, let me des- 
cribe briefly one recent project in which I was 
involved. The experiment was carried out in 
the Racine, Wisconsin, Unified School District 
under the joint sponsorship of the Wisconsin 
R & D Center and the Creative Thinking Project 
at the University of California in Berkeley 
(Olton, Wardrop, Covington, Goodwin, 
Crutchfield, Klausmeier, & Ronda, 1967). Its 
major purpose was to investigate the extent to 
which increments in the thinking and problem- 
solving performance of fifth-grade students 
could be brought about by the use of programed 
lessons (Covington, Crutchfield, & Davies, 
1966) which were designed to teach skills and 
strategies of creative thinking independent of 
any, specific subject field. Of the 47 fifth- 
grade classrooms in the school district, 44 
were used in this experiment, 3 classrooms 
having been randomly eliminated . For all 
classrooms , ratings were obtained of the ex- 
tent to which the overall classroom atmosphere 
seemed to provide an environment that facili- 
tated creative thinking on the part of the stu- 
dents. Data were also obtained on the IQs 
and achievement levels of ail students in these 
classes. The 44 classes were then divided 
into 22 matched pairs such that both classes 
in each pair had virtually the same environment 
rating and were also similar with respect to 
mean IQs and achievement test scores. One 
class from each pair was then randomly assigned 
to the experimental group (which would use The 
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Productive Thinking Program ) . The other class 
from each pair then was placed in the control 
group. A1J students were given a battery of 
tests of creative problem solving to provide 
baseline information. At the conclusion of the 
experimental training program, another similar 
battery of tests was administered. After the 
posttests were administered but before any 
scoring was done, eight males and eight fe- 
males were selected from each class as fol- 
lows: within each ciass, all male students 
were rank-ordered on the basis of IQ and one 
student randomly selected from each eighth of 
the distribution. Eight females were then se- 
lected from each class in the same way, so 
that the statistical analysis was based on the 
scores of 44 x 16 = 704 students. 

However, the analysis was carried out using 
mean scores for males and females in each 
class. We thus had a design with three factors 
(treatment group, classroom environment, and 
sex) and 11 observations (classrooms) per cell, 
not the 88 per cell we would have had using 
individual student scores as the unit of analy- 
sis . The consequence of this should be obvi- 
ous: the within-cell variance estimate in the 
analyses of variance had 80 degrees of freedom 
associated with it instead of 696. In this ex- 
periment, this was not an important limitation. 
(An F ratio of 3.96 is required for significance 
at the ,05 level with 1 and 80 degrees of free- 
dom, as contrasted with an F of 3.86 with 1 
and 696 d.f.) It is easy, however, to conceive 
of situations in which the loss of power because 
of the limited number of degrees of freedom 
would be quite severe. 

To return to the creative-thinking experi- 
ment, I would like to indicate briefly the con- 
tributions of this particular project to theory 
refinement. The results provide evidence that 
the training materials are effective in develop- 
ing creativity and problem-solving skills. In 
conjunction with other research, they indicate 
the necessity for teacher participation as an 
integral part of the training program. An im- 
portant question must be asked in view of the 
findings concerning IQ. There was a signifi- 
cant relationship found between IQ and perform- 
ance on the creativity and problem-solving 
tasks used, but no significant IQ by treatment 
interactions. It is surprising to find these 
relationships . One could readily justify ex- 
pecting just the opposite relationship (that is , 
an IQ by treatment interaction but perhaps not 



a significant relationship between IQ and 
performance on the creativity tasks) . 

The experiment just described is what 
Campbell and Stanley (1963) call design No. 

4, the Pretest-Posttest Control Group design. 

As such, it has high internal validity (freedom 
from bias). However, because of such possi- 
ble limitations to external validity as what 
Campbell and Stanley have called the inter- 
action of testing and the experimental treat- 
ments (partially controlled by using some tests 
very unlike anything in the training materials) , 
and particularly because the experimental 
sample was taken from a population which was 
limited in some ways but was in many ways 
representative of students with a wide range 
of backgrounds and abilities , any generaliza- 
tions beyond the Racine Unified School Dis- 
trict must be made with caution. In the May 
1967 American Educational Research Toumal . 
Hickrod described what he called a "metro- 
politan case study." Although the problem- 
solving research in Racine was a multiclass- 
room experiment, it is in its sampling and 
generalizability aspects much like Hickrod' s 
metropolitan case study. 

In this presentation, I have attempted to 
indicate the value of controlled experimenta- 
tion in multiclassroom settings. I have not 
yet considered the kinds of resources such 
research demands . But resources are an im- 
portant consideration. Multiclassroom experi- 
mentation places greater demands on the staff 
and requires stronger financial support than 
does most educational research. It is for this 
reason that it is most likely to develop in the 
context of large-scale activities , such as the 
major curriculum development projects , re- 
search and development centers, and perhaps 
the regional laboratories . 

Most importantly, though, such research re- 
quires the wholehearted cooperation of the 
schools . They must be willing to commit the 
time of administrators, teachers* and students. 
In order to secure such commitments, educa- 
tional researchers must be prepared to convince 
school people of the need for and particularly 
the value of the proposed research. This in turn 
requires that we give some careful thought to our 
choice of variables for study , concentrating on 
those which seem to have the greatest potential 
for significantly advancing both theoretical un- 
derstanding and classroom practice. 
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Educational researchers have long been 
plagued with the problem of finding enough 
experimental units to conduct research in the 
school setting. Typically, intact classrooms 
are available to the researcher, and the class- 
room is the only possible unit of randomization. 

In these instances the proper unit of analysis 
is the mean of each classroom. Of course, a 
large number of classrooms involving hundreds 
of children is required to provide data for a 
sensitive statistical test. 

In the event that pupils within classrooms 
may be randomly assigned to the experimental 
treatments, a different problem is encountered. 
Experimental arrangements , such as the pro- 
cess of randomization and splitting into groups, 
the use of different classrooms and strange 
teachers , may be so unusual to the pupils in- 
volved that these arrangements interact with 
the treatment. Such reactive arrangements , as 
they are termed by Campbell and Stanley (1963) , 
jeopardize the generalizability of results. 

In response to these experimental design 
problems , Research and Instruction (R & I) 

Units were conceived and established in 1966 
in elementary schools of five Wisconsin cities. 

An R & I Unit is staffed by a Unit leader, 3-6 
certified teachers, and several paraprofessionals . 
The Unit leader typically has some advanced 
training in curriculum and educational psychol- 
ogy, including methodology. This team is re- 
sponsible for the education of 100 to 200 chil- 
dren, perhaps from two or three grade levels. 
While distinctive features of this organizational 
plan incorporate the better aspects of team 
teaching, let us concentrate on the research 
capability of these Units . 

Of major importance is the flexibility which 
the Unit organization permits. Teachers and 
pupils typically change rooms as pupils are 



regrouped for instruction throughout the day. 
Assignment of pupils to new groups, instruc- 
tors, or classrooms thus is an ordinary rather 
than unusual experience. Experimental arrange- 
ments requiring random assignment of pupils to 
groups are thus far less likely to react with the 
treatment than would be the case in experiments 
involving pupils from self-contained classrooms. 

Teachers as well as pupils may be randomly 
assigned to treatments; furthermore teachers 
may be rotated among treatments so that a 
potential source of confounding is eliminated. 

In the R & I Unit staff, the experimenter has 
persons to administer the treatments who are 
both qualified to teach and have some appre- 
ciation of requirements of experimental rigor. 

Use of teachers as experimenters not only 
makes the research generalizable to usual 
school situations where children are taught by 
certified teachers; it also exposes the teachers 
to new methods , contributing to their profes- 
sional growth. Thus we may conclude that the 
R & I Unit provides not only a propitious setting 
for the conduct of educational research, in 
which several shortcomings of other arrange- 
ments are overcome, it has some beneficial 
effects as well. 

This is not to say that all the problems of 
conducting a true experiment in the school 
setting are solved. Strictly speaking, perform- 
ing the randomization on pupils is a necessary 
but not sufficient condition for their being 
treated as the unit of analysis . If the pupils 
are instructed as a group after randomization, 
then intrasession history, as well as the treat- 
ment, can affect the measurements gathered on 
a particular group. In several R & I Units last 
year, however, the experimental treatments 
were individually applied, and thus even this 
requirement for a true experiment was met. 



Let us now consider two of the many experi- 
ments which were conducted last year in R & I 
Units. The experimental treatments in each 
instance were individually applied after pupils 
were randomly assigned to treatments. Both 
experiments represent attempts to translate 
motivational theory into practice in the school 
setting. The significance level for testing each 
null hypothesis was set prior to the experiment 
at .10, a level considered appropriate for pre- 
liminary research. 

The first experiment was concerned with 
effective ways of implementing the principle 
that pupils should receive informative feedback 
relative to their school performance (Klausmeier, 
Quilling, & Wardrop, 1968). Conducted in an 
inner-city school in Racine, Wisconsin, the 
experiment was designed to assess the effect 
of feedback on acquisition of mathematics con- 
cepts. The subjects were 72 children in a 
second-grade R & I Unit classified as "disad- 
vantaged" under Title I terms. The IQs on the 
Kuhlmann -Anderson ranged from 72 to 136, with 
a mean of 101. In October of second grade, 
the mean grade equivalent score on the arith- 
metic subtest of the Stanford Achievement Test, 
Primary I battery, was 1.6. Pupils were strati- 
fied by age — younger or older — and by sex. 
Twenty-four children, six from each age — sex 
group were randomly selected as experimental 
subjects. The remainder of the group served 
as a control. 

All pupils received individualized mathe- 
matics instruction. To assist teachers in 
assessing each individual's progress, an indi- 
vidual record folder was designed. All the 
major concepts and subconcepts in mathematics 
were identified and compiled into a type of 
checklist in the individual folder. Beside 
each concept was a square to be colored in 
as an indication of the mastery of that concept. 
Diagnostic tests were developed for each main 
concept and subconcept. The format of the 
folder made it appropriate for use in parent - 
teafcher and pupil— teacher conferences. 

After each child’s initial standing was de- 
termined using diagnostic tests and teacher 
observations, instruction was prescribed at an 
appropriate level. As many as four different 
instructional groups, each focusing on a dif- 
ferent concept, were conducted simultaneously. 
Children were continually shifted among groups 
as concepts were mastered and new skills re- 
quired. When a child appeared to have per- 
sistent difficulty in grasping one concept over 
a long period of time, he was allowed to go on 
to another group and return to the troublesome 
concept later. In this way, no child met con- 
sistent frustration, and individual mobility was 
assured. 



The experimental pupils additionally met 
individually with a teacher for five minutes 
each week to discuss progress and identify 
goals for the following week. At this time the 
child was allowed to color in the square for 
the concepts acquired during the week and was 
praised and encouraged by the teacher. Each 
teacher met individually with six of the experi- 
mental students every week for 5 weeks; then 
teachers were rotated. In this way, the effect 
of a particular teacher on a child's progress 
was minimized . The duration of the experi- 
ment was 20 weeks. Each pupil, experimental 
or control, had access to his folder at all times. 

Two dependent variables were used to as- 
sess the effects of the experimental treatment 
— the number of concepts mastered (or squares 
colored in) during the experiment and the score 
obtained on a teacher -constructed posttest 
which sampled the tasks enumerated in the 
mathematics folder. The means for each group 
are presented in Table 1 . 

Table l 

Means on Dependent Variables for Assessing 
the Effect of Feedback on Acquisition of 
Mathematics Concepts 





Group 






Experimental 


Control 


Concepts 

mastered 


100 


75.0 


Score on 
posttest 


79 


63.5 



A multivariate analysis of variance was per- 
formed on the data. The results indicated that 
the treatment was significant at the .10 level. 

Since the superior performance of the ex- 
perimental group can be attributed to social 
interaction, goal setting behavior, and praise 
as well as to informative feedback, experi- 
ments have been designed this year to clarify 
the contributions of specific components of the 
treatment. In summary, the preliminary experi- 
mentation identified a motivational procedure 
which was successful in the classroom setting; 
thus encouraged, the teaching staff is ready 
to conduct more rigorous experiments which 
will further contribute to knowledge of the 
school application of theoretical constructs . 
Involvement of school personnel in generating 
the questions and executing the treatments has, 
in this instance, made possible the iterative 
experimentation so necessary to learn why a 
procedure works. In an iterative cycle of 
experimentation, the results of an initial 
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experiment are used to plan a subsequent re- 
fined experiment m which further information 
may be gained. 

A second experiment was planned to learn 
whether older children are effective models for 
younger children of the same ethnic background 
(Quilling, Cook, Wardrop, Klausmeier, Baldwin, 
& Loose, 1968). Subjects in this experiment 
were educationally disadvantaged children in 
a Milwaukee inner-city school. The majority 
of subjects were Negro and handicapped by 
such characteristics as short attention span, 
poor self-concept, inadequate motor skills, 
and low IQ. The Pintner-Cunningham Test 
administered in the fall to the first graders 
revealed a mean IQ of 81 with the range being 
from 55 to 105. 

Fifty-seven children in the first-grade R & I 
Unit were involved in the experiment. From 
these, 22 (11 boys, 11 girls) were randomly 
selected as experimental subjects to work with 
models. The remaining 35 served as the con- 
trol group. Similarly, 22 sixth graders (11 boys, 
1 1 girls) were selected from the group of sixth 
graders nominated by their teachers to be appro- 
priate models. The older pupils came from the 
same physical and socioeconomic environment 
as the primary children . 

Each sixth-grade "helper’ 1 was randomly 
assigned to an experimental subject of the 
same sex. The older-younger pairs so formed 
spent one-half hour per week together playinc. 
arithmetic games, manipulating concrete ob- 
jects, and helping with problem solving in 
an Instructional Materials Center environment. 

In order to minimize the disruption of schedules , 
the models reported at times most convenient 
to their individual timetables. Additionally, 
the sixth graders were briefed for half an hour 
each week on appropriate activities for the 
younger children and reinforcement procedures. 
The three primary teachers each took responsi- 
bility for supervising the experimental treatment 
on a rotating basis. 

All primary children received the same mathe- 
matics instruction four days a week. A televised 
mathematics program. Patterns in Arithmetic for 
Grade 1 , was used together with an accompany- 
ing manual and teacher-made worksheets. Large 
and small group instruction was used as appro- 
priate. In addition, individual instruction was 
given to pupils when considered necessary. 

A teacher-constructed test was designed to 
measure acquisition of the mathematics concepts 
presented during the 15-week experiment. The 
test was administered to all first graders at the 
conclusions of the experiment. 

The means of experimental boys and experi- 
mental girls were higher than those of control 
groups of each sex, as Table 2 indicates. 



Table 2 



Means on Teacher-Constructed Test to Assess 
the Effect of Models on Learning 
Mathematics Concepts 



Group 




Boys 


Girls Mean 


Experimental 


16.00 


21.27 18.63 


Control 




13.78 


17.84 15.80 


The analysis of variance performed on the 
scores indicated that both treatment and sex 
were significant sources of variation. The 
statistical analysis is presented in Table 3. 
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Analysis of Variance on Scores of 
Teacher-Constructed Test 


Source 


df 


MS 


F 


Treat- 
ments (T) 


1 


108.0886 


3.3822 (p < .0716) 


Sex (S) 


1 


295.9425 


9.2604 (p < .0037) 


TS 


1 


5.0827 


.1590 


Error 


53 


31.9015 





Evaluation of the experiment also indicated 
that all experimental pairs were not compatible. 
Thus experiments in which the pairings are maxi- 
mized for compatibility and model-effectiveness 
are a logical outgrowth of the one reported here. 

Experience to date indicates that given sig- 
nificant results, the teaching staffs are eager 
to incorporate the treatment into the instruc- 
tional program or to investigate further the 
effectiveness of the procedure. The interplay 
of ideas of the classroom teacher and R & D 
specialist— be he an educational psychologist, 
a curriculum expert , or a research methodolo- 
gist — have resulted in empirical evidence of 
the applicability in the school setting of prin- 
ciples derived from motivational theory. That 
instruction has been improved simultaneously 
is evident from the field testing results of 
schools with R & I Units (Klausmeier et al., 
1968; Quilling et al. , 1968). 

In summary, the R & I Unit is so organized 
that the conduct of true experiments in the 
school setting is possible. Staffing and 
physical arrangements allow random assign- 
ment and individual treatment of pupils . Ex- 
emplars of such experiments , in which moti- 
vational theory was translated into practice, 
were given. Of promise for the future is the 
climate for iterative experimentation which has 
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been created. Already results from initial ex- 
periments have been used to design subsequent 



experiments by which teaching-learning theory 
may be further refined. 
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The purpose of this paper is to describe a 
major activity of a number of projects within 
Program 2 of the R & D Center, namely the 
development and »ofincmcnt of prototypic in- 
structional systems. 

Program 2, Processes and Programs of In- 
struction, is one of three major programs of 
the Center attempting to improve educational 
practice utilizing knowledge about cognitive 
processes and applying this knowledge to in- 
structional problems. The program has five 
general objectives: 

1) To establish rationale and strategy for 
developing instructional systems in the 
cognitive domain. 

2) To identify by careful synthesis and 
further research sequences of concepts 
and cognitive skills within and across 
disciplines . 

3) To develop assessment procedures and 
materials for the concepts and skills 
identified above. 

4) To identify existing materials or develop 
new instructional materials associated 
with the concepts and cognitive skills. 

5) To generate new knowledge through re- 
search about instructional procedures 
including motivation, individualization, 
classroom management, and organization 
of instruction. 

In order to reach these objectives, instructional 
programs in mathematics, English, reading, 
speech, and science are currently being studied. 
One aspect of this overall effort is the develop- 
ment of prototypic systems. 

The development of prototypic instructional 
systems involves an attempt to spell out and 
validate operational plans and specifications 
for instructional systems which have certain 



components. Prototypic, stemming directly 
from prototype, implies that what is to be 
developed is intended to be not a fully devel- 
oped program, but guidelines or specifications 
for complete programs. The word was chosen 
to convey the fact that we are neither develop- 
ing complete instructional programs or mate- 
rials nor competing with large curriculum proj- 
ects. A prototypic instructional system is 
only intended to be a framework for further 
development. 

The strategy being followed is quite differ- 
ent from that used by curriculum projects such 
as SMSG or PSSC. None of those curriculum 
projects attempted to change in any way the 
basic educational units of our school system. 
The, ,nly attempted to change the content of 
physics or mathematics or chemistry within 
the existing school organization. For example, 
algebra has been traditionally taught one hour 
a day five days a week in groups of 30 with 
one teacher. Developers of new programs 
made no attempt to change that pattern; in- 
stead , they worked within the existing struc- 
ture of most American schools and restricted 
the revision to subject matter. 

However, some schools today are now, and 
most schools of some not too distant tomorrow 
will be, quite different from the stereotype 
school those curricula were developed for. 

With the advent of flexibly organized schools, 
such as the Multiunit School, more carefully 
engineered instructional programs must be 
invented . 

One common basic function of contemporary 
elementary schools is "to execute a system- 
wide standard instructional program designed 
by others [Klausmeier, Morrow, & Walter, 

19 68, p. 1]. M On the other hand the school 
of the future will be "developing and executing 



an effective system of individually guided edu- 
cation within each building [Klausmeier et al., 
1968, p. 1].” Thus, the long unrealized goal 
of providing instruction that is effectively 
geared to the individual learner is now possible. 
The intent of developing prototypic instructional 
systems is to construct the specifications for 
individually auided instructional programs . 

The strategy being used has been (1) to 
identify component attributes of an instructional 
system which might be manipulated or changed, 
and (2) to construct sample elements for an 
instructional system which can operate within 
a flexible educational system. 

The word system as used in the paper refers 
to a man-made controlled functional structure. 
Man-made structure means that the system has 
interdependent components which can be changed 
or manipulated. Controlled means that there is 
a feedback or monitoring procedure which can 
be used to manage the system, and functional 
means that the system is goal oriented with a 
stated purpose or intent. Minimally such a 
system has four basic components: input, 
mechanism, feedback and output. The input 
is raw materials: the mechanism is the way in 
which the raw materials are transformed into 
the output; the feedback is the control process; 
and the output is some product reflecting the 
purpose of the system. 

In an instructional system the input is stu- 
dents; the mechanism is the instructional pro- 



gram; the feedback is an evaluation program: 
the output is the same students with changed 
behaviors. There is also, m an instructional 
system, a resource component which includes 
instructional materials and equipment, teachers 
and staff, and facilities and their allocation. 
Each component could be viewed as a subsystem 
with internal, interdependent elements. Three 
elements of the evaluation component — moni- 
toring procedures for the input, the output and 
the resources of the system; a data collection 
and management element; and decision-making 
procedures in which criteria and explicit de- 
cision rules are established — are identified in 
Figure 4 , and their relationship to each other 
and to other components is illustrated. Thus, 
the development of prototypic instructional 
systdhis involves detailing the design specifi- 
cations for each component and the relation- 
ships of the components. 

In developing operational plans for an in- 
structional system, activities are carried out 
to answer a number of queries. Why? requires 
the identification of purposes and goals. 
What? is answered by an outline of content. 

To whom? necessitates consideration of per- 
formance history, o^ganismic variables, and 
cultural influences. When? demands specifi- 
cation of time allocation and sequencing of 
the content elements. Specification of trans- 
mission procedures answers How? Does it 
work ? is answered by the development of 




Figure 4. Model of an Instructional System 
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measurement tools and of evaluation and 
decision-making procedures. 

Goals are statements of intent which give 
meaning to content selection. They may bo 
broadly conceived for the whole system, as 
utilitarian or cultural goals whose establish- 
ment is a societal responsibility. Or goals 
may be more narrowly conceived in terms of 
basic intellectual disciplines. Separate in- 
structional systems can be developed for read- 
ing, mathematics, science, etc. as subsystems 
of a total educational system. This second 
type of goal is the focus of Program 2 of the 
Center for disciplines with high cognitive con- 
tent and agreed upon cultural value. Proto- 
typic instructional programs for the elementary 
school are currently being developed in reading, 
science, and mathematics . Similar programs 
in language arts and speech are being worked 
on at higher grades . 

Initial conceptualization of a content-based 
instructional system is done by outlining the 
content to include concepts (facts, assump- 
tions, definitions, symbols, etc.), principles 
(associations between facts), conceptual 
schemata or structure (the organization of a 
set of facts and principles) , and methods (the 
ways in which scholars use facts, principles, 
conceptual schemata, and skills). But an 
outline is not sufficient. Descriptions of ob- 
servable behavioral characteristics for students 
who have acquired each concept or mastered 
each skill must be prepared. The translation 
of content into behavioral objectives or de- 
sired terminal behaviors is critical since state- 
ments of observable performance give direction 
to the development of evaluation procedures. 

Individual differences between learners are 
well documented. However, only a small part 
of this knowledge has be-. n related to instruc- 
tional programs . The abilities of learners are 
dependent upon their de eiopmental level, their 
intellectual power, tl.eir learning style, their 
past learning experience, their parental rela- 
tionships, their peer group associations, etc. 

In addition, the emerging picture indicate" that 
perform an e history (the nature and quality of 
prior c xences) in a subject area is particu- 
larly important. One of the critical unsolved 
engineering problems in the construction of 
individualized instructional programs is how 
one can incorporate this wealth of information 
about individual differences into the system so 
it can be used to guide an individual's learning 
experiences. This information needs to be in- 
cluded as one considers each step in the devel- 
opment of operational plans. 

Once the content domain has been specified, 
the next task is to begin to put the concepts 
and cognitive skills together in some sequence 



and to allocate time, age level, number of 
days, etc. which will be assigned to this 
concept or that skill. This is not an easy 
task. It involves determining prerequisite 
behaviors or prerequisite learned concepts 
and skills. For each skill it involves ex- 
amining pupils and their general cognitive 
development, and it involves consideration 
of alternative sequencing based on perform- 
ance histories. Clearly' there is no one best 
sequence. Alternative sequences and time 
allocations must be developed. 

Given an outline of content and cognitive 
skills and a sequence and time allocation of 
these skills for an instructional program, the 
details of an instructional program can be des- 
cribed by a list of the ways in which these 
concepts and skills can be transmitted to stu- 
dents. This involves two steps: first, des- 
cribing potential student activities which, if 
followed, reach desired objectives; and sec- 
ond, specifying what resources arc to be used 
and how they are to be used. The focus of 
how information is to be transmitted to students 
has to be on the activities he is asked to per- 
form. The character of the activity is depend- 
ent upon both what is to be transmitted and the 
personality of the student. Again alternative 
activities must be invented. 

Resou.ce allocation involves detailing the 
teacher's activities in organizing and directing 
the student activities. It also involves speci- 
fying what instructional materials and/or sup- 
plies are to be used. And, it further means 
that the organization for instruction — i.e. , 
nongraded, self-contained, unrt or team; one 
subject, more than one subject; modular time 
schedule, etc. — be detailed. 

Concurrently, measurement tools and eval- 
uation procedures need to be developed to 
measure students' prior achievements or readi- 
ness to engage in the program, to measure the 
students’ more general aptitudes and charac- 
teristics including their social background 
characteristics, to measure progress during 
short and long intervals of instruction, and to 
evaluate the components and total system in- 
cluding student progress and level of achieve- 
ment. To do all this involves spelling out 
what is to be monitored with respect to initial 
behavior, terminal behavior, and resource 
allocation; how these data are to be collected 
and managed; and what kinds of decisions are 
to be made on the basis of these data. 

Our long-range plans require consideration 
of each area mentioned. However, for short- 
range planning emphases are only on selected 
components. Projects in reading , science and 
mathematics are described below to illustrate 
what is now being done. 
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In reading. Professor Wayne Otto and his 
staff working with school personnel have identi- 
fied reading skills and assessment procedures 
(Otto, Saeman, Houston, McMahan, SWcita 1 , 
1967). The intent was; 

to go beyond the simple sequencing of 
skills by attempting to suggest specific 
exercises and/or observations designed 
for use in assessing mastery or progress 
in the development of each of the sev- 
eral skills. Concurrently, an attempt is 
made to avoid limiting the use of the 
Guide to a particular instructional setup, 
e.g., ungraded, self-contained, inter- 
class, etc. The specific suggestions 
for assessing skill development should 
provide not only a means for gathering 
information but also a guide to system- 
atic teaching of skills. 

Current work in the reading project is to 
refine and clarify these skills and assessment 
procedures and to extend the Guide to include 
a catalogue of materials which can be used to 
teach each skill. 

A second example is the science project 
headed by Professors M. O. Pella and G. T. 

O' Hearn. Science, in contrast with reading, 
has not normally been taught in elementary 
schools, hence, there is no accepted set of 
concepts and skills. Identifying these content 
elements and seeing whether young children 
can acquire them is a major part of their current 
efforts. In addition, they are engaged in estab- 
lishing behavioral objectives and developing 
related assessment tools. 

In mathematics, another approach is being 
taken by Professor H. J. Fletcher and myself. 
We began with broad mathematical objectives 
d then identified the concepts, skills and 
operations {basic units) involved in reaching 
these objectives. After the basic units have 
been compiled, the following steps are taken: 

a) Each unit or concept is analyzed in terms 
of its subconcepts, properties, or attri- 
butes, together with any rules necessary 
for combining them. 

b) Prerequisite behaviors the student must 
possess for any unit are identified. 

c) These prerequisite behaviors are used to 
develop a logical sequencing of the units. 

d) Student behaviors are associated with 
each unit. Observation of these behav- 
iors provides the only method for evalu- 
ating change in student behaviors. 

e) Teaching tactics are identified for each 
unit. Tactics include determining whether 
the student has the necessary prerequi- 
sites for a unit, assembling the necessary 



material, performing demonstrations, 
posinq problems, etc. 

f) Problem situations are designed to secure 
and hold attention, or to maintain high 

motivation . 

The above steps outline the conceptual 
analysis phase of our project. A pilot exam- 
ination phase follows and consists primarily 
of exploratory pilot research in which each 
problem situation is tried out with a few stu- 
dents in a normal classroom. In a subsequent 
evaluation phase the effectiveness of each 
problem situation is assessed. Unfavorable 
results would necessitate returning to an earlier 
phase; favorable results permit continuing into 
the next phase, validation, in which careful 
studies are conducted on all elements of the 
system prior to field testing of the entire system. 

What these examples imply is that we are 
trying to conduct research-based development . 
The problem, although initially an instructional 
development problem, must be conceptualized 
in terms of knowledge about learning (concept 
formation, rule learning, problem solving, 
motivation, etc.). The knowledge and methods 
of the science of psychology must be brought 
to bear on instructional problems. But the 
connection goes both ways . Many real instruc- 
tional problems have not been adequately 
studied within a theoretical or laboratory 
framework. It is now becoming apparent that 
much basic research will be development- 
oriented research . 

To construct a prototypic instructional sys- 
tem real interdisciplinary scholar- practitioner 
teams must be working together on the problem. 

In the mathematics project mentioned above 
a psychologist, a mathematician, a mathe- 
matics educator, a classroom teacher, and 
five research assistants (in both mathematics 
and psychology) are involved. One must also 
have an appropriate environment in which to 
try out one's ideas. The Multiunit School 
developed by Professor Klausmeier and his 
staff at the R & D Center is proving to be an 
excellent environment in which to do program- 
atic research. In addition, one must have the 
luxury of time to do long-range program atic 
research. The R & D Center concept provides 
for long-term projects aimed at major conceptual 
and programatic problems. 

In summary, prototypic systems of instruc- 
tion are operational plans for educational sys- 
tems which have been carefully formulated and 
validated. We believe that the development 
and refinement of prototypic instructional sys- 
tems as being done at the Wisconsin R & D 
Center will significantly influence American 
education in the future. 
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V 

EXPERIMENTATION RELATING TO FORMATIVE EVALUATION 



Richard E. Schutz 

Director. Southwest Regional Laboratory 



1 am going to restrict my remarks to one- 
half of the title of the symposium. I will be 
talking about development strategies and in- 
structional improvement and will not attempt 
to deal with research or theory refinement. 
Research and theory go together like develop- 
ment and improvement. The distinction is 
primarily in terms of the outcomes generated. 
Research produces refined knowledge; develop- 
ment produces usable products. Research 
seeks the answer to the question How now? 
Development seeks the answer to the question 
How to ? 

Scriven (Tyler, Gagne, & Scriven, 1967), 
you will recall, introduced the term formative 
evaluation to describe the evaluation of edu- 
cational programs that are still in some stage 
of development. Formative evaluation con- 
trasts with summative evaluation, the evalua- 
tion of finally developed educational programs. 
The product of formative evaluation activities 
is expected to be an improved instructional 
program, while the product of summative eval- 
uation is normally a set of descriptive state- 
ments about a single program or about the 
relative merits of two or more programs. An 
extensive methodology is associated with sum- 
mative evaluation, since researchers have had 
extensive experience in comparing two or more 
existing programs. Formative evaluation, how- 
ever, is a different matter. 

At present, formative evaluation methods 
have much the same status as the invisible 
needle and thread used by the tailors in the 
"Emperor's New Clothes." For a variety of 
social, political, and economic reasons , edu- 
cational researchers have derived satisfaction 
in viewing their efforts as contributing to edu- 
cational improvement. But the inescapable 
fact is that reliable ways of effecting educa- 



tional improvement have yet to be identified 
either by researchers, manufacturers, or school 
personnel. 

J. M. Stephens, in his highly ptovoeative 
little book The Process of Schooling (1967) , 
summarizes the summar.es of experiments on 
instruction over the last 50 years. This is a 
disturbing treatise which I highly recommend. 
Stephens comments uoon "the remarkable con- 
stancy of educational results in the face of 
widely differing deliberate approaches." He 
goes on to state bluntly, “ Every so often we 
adopt new aporoaches or new methodologies 
and place our reliance on new panacea* . At 
the very least we seem to chorus new slogans. 
Yet the academic growth within the classroom 
continues at about the same rate, stubbornly 
refusing to cooperate with the bright new dicta 
emanating from the conference room [p. 9].” 

A spectacularly elegant NSD. 

Stephens' prescription is to relax and enjoy 
the powerful and pervasive educational forces 
which apparently work well apart from any de- 
liberate direction. Get a few adults, put them 
in contact with kids for as much time as pos- 
sible, and count on spontaneous factors to 
take care of everything else. 

i find this line of reasoning a very threaten- 
ing but compelling argument, things being what 
they are . The aim of development, however, 
is to insure that things do not remain as tney 
are. Development seeks to produce materials 
and methods called products which can be used 
to produce a specified outcome. That, the tie- 
sired outcome can be produced is alwavs a 
matter of faith while the development is in 
progress. There is no guarantee that we will 
ever place a man on the moon. Neither -g 
there any guarantee that we will ever teach 
all children to read. But in each example. 
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there is a high probat llity that the investment 
ot cumulative effort over time will achieve the 
desired end. 

.Educational development is at present in a 
state of conceptual deprivation. All of the 
terms available for describing instruction have 
reference to instructional operation systems — 
Keeping school — rather than to instructional 
development. For example, grade and subject- 
matter dimensions and puoil characteristics are 
useful in defining school operations. To indi- 
cate that someone is producing a high school 
American history program, for example, appears 
to communicate a good deal. However, the 
information relates almost exclusively to use 
of the instruction in the schools. It indicates 
very little about the instruction per se and in- 
dicates nothing about the operations involved 
iii producing the instructional materials and 
procedures . 

The principal dimension traditionally manip- 
ulated in instructional development has been 
subject-matter content- a la the new math, 
science, and social studies "curriculum devel- 
opment" projects. Formative evaluation efforts 
have been highly content oriented. Measure- 
ment results have been used as a basis for 
making decisions dealing with the sequencing 
o. f substantive concepts and for modifying the 
instructional verbage associated with a given 
concept . When the measurement results are 
not easily interpreted, one of two events takes 
place. (1) The proposed outcomes of the in- 
struction are modified, or (2) the goals are re- 
defined in mere general or abstract terms to 
facilitate agreement about the interpretations 
of the measurement efforts. But substitutions 
in instructional goals should not be confused 
with improved attainment of these goals. Nor 
is it defensible to retreat behind a mystique of 
complicated and intangible criteria. 

If one takes a product-oriented approach, 
the criteria for gauging instructional improve- 
ment become straightforward and the applica- 
tion of formative evaluation procedures more 
meaningful. Remember that we have defined 
a oroduct as consisting of the materials and 
accompanying procedures to accomplish speci- 
fied instructional outcomes. In this setting 
we are concerned with three criteria dimensions: 
reliability, utility, and cost. 

We must be concerned first with the relia- 
bility of the product in accomplishing the ob- 
jective. How well does it work? Here we are 
considering such things as dependability of 
the outcome, yeneralizabiiity or exportability 
to a variety of locales, and replicability. Until 
an effect can be produced with an acceptable 
degree of dependability, there is really nothing 
for a potential user to evaluate. Thus reliability 
of effect is an important criterion. 



A second set of criteria involves utility . 

How useful is the result? Here we can dis- 
tinguish between social utility, which involves 
an outcome that is useful at present, and in- 
structional utility which involves an outcome 
that provides a base for future educational 
attainment. To be useful, a given outcome 
must have clearly demonstrable social and 'or 
instructional utility. 

When we are able to produce an instructional 
effect with a given degree of dependability, and 
it is judged to be worth producing, then we must 
necessarily be concerned about cost . Here we 
are talking about either the time or the money 
required to produce a given result. Any form 
of cost can be transformed into one of these 
two dimensions . 

It is only when one is able to produce use- 
ful outcomes with a reasonable degree of relia- 
bility that it makes any sense to talk about 
cost benefit analysis. But with reliability, 
utility, and cost targeted you can very reason- 
ably talk about both cost efficiency and cost 
effectiveness. Cost efficiency involves the 
ratio of reliability to cost; cost effectiveness 
involves the ratio of both reliability and utility 
to cost. 

Education has yet to cope adequately with 
the first set of criteria — reliability of effect. 

We simply lack the wherewithal at this time 
to dependably accomplish the attainment of 
the educational outcomes for which the schools 
have assumed responsibility. At present we 
can at best plan strategies which have some 
likelihood of leading to the improvement de- 
sired. Several general alternative strategies 
can be formulated which themselves will pro- 
vide the basis for a grand experiment. I shall 
limit my considerations to the experimentation 
possible within a single such general strategy. 

The strategy is derived by analogy from 
engineering; underscore analogy. Analogy is 
the weakest form of argument, but the relation- 
ship at present between educational and engi- 
neering development is analogous at best. 

The foundation of the strategy is planned 
iteration — a set of sequenced and coordinated 
activities which through careful management 
cumulate to effect the improvement desired. 
That's fine. But, what do we iterate? Or, 
what should we iterate? Well, we have to 
start with what we've got. Here we can identify 
several manipulable dimensions of instruction. 
For example: 

The instructional media — at present 
this is typically the teacher or printed 
matter, although a wide spectrum of 
auditory and visual presentation mech- 
anisms are possible. 

The pacing mode — at present in- 
struction is group paced, although 
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various degrees of differentiation 
toward individual pacing can be identi- 
fied . 

The monitoring contingency — at 
present the attainment of instructional 
outcomes is norm-referenced, if con- 
tigent at all. That is, one lesson 
typically follows another, irrespective 
of whether ail learners have attained 
the outcomes the lesson was designed 
to produce. 

The final example of possible dimen- 
sions is the monitoring agent. At pres- 
ent, this is the teacher. The teacher is 
responsible for insuring that the instruc- 
tion is sequenced, coordinated with 
other activities. But the pupil himself 
could make a contribution in the area as 
could other pupils, parents, computers, 
etc. 

Describing current school practices using 
the four dimensions of 0) Instructional Media; 
{2) Pacing Mode; (3) Monitoring Contingency; 
and (4) Monitoring Agent, formal public edu- 
cation at the present time can be characterized 
as print mediated, group paced, managed 
against relative norms, and teacher-based. 

There is good reason to believe that instruc- 
tion which is multiple mediated, individually 
paced, managed against objectives, and com- 
puter-based would be much more effective. 

My hunch is that it will be at least a decade 
before such a system is operationally feasible. 
But this hinges on the referents for the term 
feasible. Prom a researcher's point of view, 
any phenomenon which can be conceptualized 
and potentially manipulated is regarded as 
feasible. If the development does not occur 
easily, the researcher honestly believes that 
the persons reponsible for translating his 
efforts into usable instructional products and 
operating the instructional program have hope- 
lessly fouled-up the concepts. The developer, 
on the other hand, finds himself in an awkward 
"Mr. In-between" role. He views the research- 
er's ideas as perpetually incomplete and inade- 
quate as a development guide. At the same 
time, he necessarily views current school 
operations as obsolete since his efforts are 
directed toward improving it. School personnel, 
for their part, forever find the present work of 
both the researcher and the developer unrelated 
to immediate everyday problems. Each is accu- 
rate. Moreover, this is not an undesirable 
state of affairs so long as it is recognized; it 
provides the basis for an efficient division of 
professional labor. 

How is this educational improvement to be 
effected? I should like to suggest two concur- 
rent and compatible tactical approaches. Each 
recognizes the present print-mediated, group 



paced, weakly managed , and teacher-based 
instructional system. One approach involves 
a series of straightforward trial-revision 
cycles to sharpen up the accomplishment of 
given instructional outcomes, working within 
the boundaries of the present instructional 
system. An analogy is provided by the modi- 
fications in the Volkswagen annually to im- 
prove its performance in specified ways . The 
second approach involves a major manipulation 
of a system dimension per se which produces 
a new generation of the product. Following 
through on the automobile analogy, this might 
be an electric car. 

Each of these two approaches involves a 
convergent iterative methodology. Let me 
mention the sequence we find useful at the 
Southwest Regional Laboratory in pursuing the 
first approach. It involves first the prepara- 
tion of instructional specifications . This is 
a set of sequenced statements of desired in- 
structional outcomes accompanied by specifi- 
cations of requisite entry behavior, learner 
activities, specific conditions under which 
practice must be given to assure that appro- 
priate behavior does and does not occur, and 
specifications for testing if the outcome is 
attained. 

Component preparation and tryout involves 
the initial production and tryout of component 
methods and materials which relate to various 
aspects of the instructional specifications. 
These tryouts involve "mock-ups" designed 
to reflect the critical features of the aspect of 
instruction about which further information is 
needed before it can be considered a reliable 
prototype component. 

Product preparation and tryout involves com- 
bining and extending the prototype components 
into instruction suitable for classroom use. 

The objective is to produce methods and mate- 
rials which are attractive to pupils, manage- 
able by teachers , at as low a cost and requir- 
as little instructional time as possible. The 
product tryout is conducted by regular school 
personnel under standard school conditions. 

Each of these stages involves a continuous 
sequence of trial-revision iterations to suc- 
cessively eliminate the defects in the product 
and to increase its effectiveness. The se- 
quenced instructional objectives provide the 
criterion against which base-rate performance 
data can be generated. This provides an assur- 
ance that the changes introduced cumulate in 
improvement and avoids expensive unevaluated 
modifications which fail to improve overall per- 
formance . 

The principles of scientific experimentation 
— for example those discussed by Campbell 
and Stanley (19631 under the rubrics internal 
and external validity — are highly relevant at 
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each stage of the cycle. But, development is 
concerned with the management of variance 
rather than the analysis of variance. The com- 
parisons of interest involve variations and re- 
finements introduced sequentially over time 
rather than concurrently at a single time. Note 
that I'm not talking here of a time-series de- 
sign in the Campbell-Stanlev sense. Treat- 
ments Aj , A2, etc. are compared sequentially 
in toto as each treatment is successively re- 
viewed, refined, and recycled on fresh samples. 
Treatment A2 always incorporates "the best" 
of Ai; A3 the best of A2; and so forth. If a 
given iteration yields worse rather than better 
results, one goes back to the precondition and 
reformulates the modification to be made. Our 
experience to date has been that this iterative 
approach does pay off. 

So much for the first approach, effecting 
cycles of improvement. In effecting genera- 
tions of improvement one is seeking to intro- 
duce a modification in at least one system 
dimension with the objective of concomitantly 
increasing effectiveness along other dimensions. 
This involves complex development work that is 
completely removed from the refinement cycles 
within a present generation. 

Consider the instructional monitoring dimen- 
sions. The Southwest Regional Laboratory, in 
conjunction with System Development Corpora- 
tion, is currently engaged in the development 
of a computer-based instructional management 
system. Here the immediate concern is not 
with all instructional functions , but simply in 
developing workable procedures to provide the 
teacher with frequent information concerning the 
progress of each student with respect to the in- 
structional outcomes of interest to the teacher 
together with suggestions for activities which 
are consistent with the student's performance. 



Thus in contrast to computer-assisted in- 
struction system, we are seeking a technologi- 
cally less sophisticated but more feasible 
computer-managed instruction system. The 
computer technology problems are being solved 
using the same iterative strategy outline above. 
However, rather than pupil performance, the 
criteria involve teacher management perform- 
ance. Although live classes are being used in 
developing the system, there is no expectation 
that pupil performance will be improved at this 
point. However, when the system is function- 
ing adequately, the greater monitoring capa- 
bility it will provide is expected to improve 
performance sufficiently to introduce a new 
generation of instructional effectiveness. 
Analogous programatic efforts in media and 
pacing are also being conducted by the Lab- 
oratory. 

In sum, educational development appears 
to be as vast and promising an enterprise as 
nuclear or space development. Just as engi- 
neering enterprises find it necessary to use 
all existing relevant knowledge plus, so does 
educational development. With this effort the 
concept of formative evaluation will be clothed 
with a technology of instructional product de- 
velopment. This is at once a challenging and 
threatening enterprise for those who become 
involved in it. The "produce or perish" impera- 
tive associated with educational development 
is an even tougher task master than the "pub- 
lish or perish" imperative of academia and the 
"profit or perish" imperative of business. 
However, it is an imperative which the educa- 
tion profession can ill afford to ignore, con- 
sidering the tenor of the times. The frontier 
is there, with lots of wide open territory to be 
pioneered. If you are bright and brave, blast 
into it. 
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VI 

REMARKS FOLLOWING THE PRESENTATION OF PAPERS 
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Leslie D. McLean 

Associate Professor and Chairman. Department of Computer Applications 
The Ontario Institute for Studies in Education 



The topic of this symposium, as stated by 
Professor Klausmeier in his first sentence, is 
an important and vital one. The papers deal 
with a specification of the question, How do 
we organize for research and development in 
education? It is refreshing to find the pres- 
entation and discussion of efforts which have 
already been tried as opposed to suggestions 
of what might possibly work. It is probably 
no accident that the papers are organized 
along the lines of a research project report. 
Klausmeier and Romberg provide the theory and 
background of the problem, Wardrop and Quil- 
ling the procedures and results, and Schutz 
the discussion and critique. It has been a 
most stimulating and worthwhile symposium 
and I am pleased and honored to be a part of it. 

My first reaction is that we are seeing a 
new stage in the evolution of experimental 
design. First there was the movement of de- 
sign concepts into educational research text- 
books. Their actual use by experimenters has 
been very limited; until recently status studies 
rather than the controlled experiment were a 
mark of educational research. The change 
seems all the less surprising when we look at 
the magnitude of the effort recorded by the re- 
search and development centers. It is clear 
that a sustained series of meaningful experi- 
ments carried on in school settings does re- 
quire the trained manpower and very consider- 
able resources which the R & D centers can 
provide. The continuity in an R & D center 
can provide the support for an iterative program 
of experimentation where insights and hypothe- 
ses discovered in one experiment can be fol- 
lowed up soon enough to get answers and move 
on to the new hypotheses so generated. The 
same mistakes need not be repeated quite so 
often. 



It is interesting to compare the R & D cen- 
ters with the Ontario Institute for Studies in 
Education. The OISE was founded as an in- 
dependent college by the Province of Ontario 
with the task of carrying on research, develop- 
ment, and graduate instruction. Not only is 
the teaching dimension added, but certain as- 
pects of development and dissemination which 
have been assigned to the regional laboratories 
in the United States. Perhaps you can see that 
the topic of this symposium has direct rele- 
vance to the problems being encountered in 
Ontario . 

I am pleased to have reinforced a vague 
hypothesis which I formulated recently, namely 
that not only does one follow the path from re- 
search to development to dissemination, but 
development also produces research ideas and 
pushes one straight back to the important 
questions. As we make serious efforts to de- 
velop worthwhile instructional products, we 
are constantly confronted with unsolved re- 
search questions and constantly challenged 
to think more clearly about exactly what we 
are trying to do. Schutz' paper is a most val- 
uable contribution since few places are actually 
producing instructional products in an atmos- 
phere of research and development. 

The phrase "systems approach to education" 
has been accompanied by so much talk and so 
little action that I have developed a strong 
avoidance response to it. I must admit that 
this response is near extinction after listen- 
ing to Wardrop' s and Quilling's papers, fol- 
lowed by Romberg's paper. Not only are there 
systems on paper, but there are actually things 
called Research and Instruction Units. People 
are doing research within an overall framework 
with some attempt at coordination. I must 
again point to the OISE as an example of a 
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similar attempt; we should be able to increase 
the probability of our success given useful in- 
puts such as these. 

My final comment stems from a general 
impression I have that everything is arranged 
and prescribed for the student. The language 
we use constantly reinforces the idea that 
"we" know what is good for "them." My feel- 
ing is that this is not so much so as we might 
think it is so, but that we should make a con- 



certed attempt to develop an attitude that the 
student must begin to exercise more control 
over his own education. If I am not mistaken, 
Schutz was the only one to actually mention 
students and their place in the system. Com- 
plete control to achieve specified learning 
outcome will be quite a failure if the student 
develops no initiative of his own. Let us by 
all means carefully consider what we will call 
success . 
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